Control system and method using in-vehicle gesture input

ABSTRACT

Provided are a control system and method using an in-vehicle gesture input, and more particularly, a system for receiving an occupant&#39;s gesture and controlling the execution of vehicle functions. The control system using an in-vehicle gesture input includes an input unit configured to receive a user&#39;s gesture, a memory configured to store a control program using an in-vehicle gesture input therein, and a processor configured to execute the control program. The processor transmits a command for executing a function corresponding to a gesture according to a usage pattern.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation Application of prior application Ser.No. 16/910,633 (allowed), filed Jun. 24, 2020, which claims priorityunder 35 U.S.C. § 119 to Korean Patent Application Nos. 2019-0075605,2019-0075606, 2019-0075607, and 2019-0075608 filed on Jun. 25, 2019, inthe Korean Intellectual Property Office, the disclosures of which areincorporated herein by reference for all purposes.

BACKGROUND 1. Field of the Invention

The present invention relates to a control system and method using anin-vehicle gesture input, and more particularly, to a system and methodfor receiving an occupant's gesture and controlling the execution ofvehicle functions.

2. Discussion of Related Art

Vehicles provide additional functions for user convenience such asaudio, video, navigation, air conditioning control, and lighting controlin addition to a basic driving function.

In order to perform these functions, vehicles receive an action commandfrom a user through a button input, a touchscreen input, a gesturerecognition, or the like.

A vehicle gesture recognition system is defined as an electronic systemthat collects data with machine vision and manipulates variousin-vehicle driver support functions through an in-vehicle human-machineinterface.

However, conventional vehicle gesture recognition technology merelyreceives a gesture and provides a function corresponding to the gesture.Therefore, drivers should learn each gesture to execute a correspondingfunction (e.g., to open and raise a hand up so as to increase thetemperature of an air conditioner, to spread the index finger and themiddle finger and raise the hand up so as to raise the volume of theaudio, etc.), user convenience is low, and also it is not possible toperform different input control for the same gesture depending on thesituation.

SUMMARY OF THE INVENTION

The present invention has been proposed to solve the above-mentionedproblems and is directed to providing a control system and method usingan in-vehicle gesture input, the control system and method interpretinga gesture input in consideration of gesture input situation informationand executing a preset function corresponding to the gesture inputaccording to a result of the interpretation.

According to an aspect of the present invention, there is provided acontrol system using an in-vehicle gesture input, the control systemincluding an input unit configured to receive a user's gesture, a memoryconfigured to store a control program using an in-vehicle gesture inputtherein, and a processor configured to execute the control program. Theprocessor transmits a command for executing a function corresponding toa gesture according to a usage pattern.

According to another aspect of the present invention, there is provideda control method using an in-vehicle gesture input, the control methodincluding (a) receiving operation state information and a gesture, (b)checking whether the operation state information satisfies a presetcondition, and (c) providing different functions for the gestureaccording to whether the preset condition is satisfied.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a control system using an in-vehiclegesture input according to an embodiment of the present invention.

FIG. 2 is an exemplary diagram of a contact swipe gesture input in acontrol system using an in-vehicle gesture input according to anembodiment of the present invention.

FIG. 3 is an exemplary diagram of a non-contact swipe gesture input in acontrol system using an in-vehicle gesture input according to anembodiment of the present invention.

FIG. 4 is a block diagram showing a control method using an in-vehiclegesture input according to an embodiment of the present invention.

FIG. 5 is a block diagram of a control system using an in-vehiclegesture input according to another embodiment of the present invention.

FIG. 6 is an exemplary diagram showing a gesture input controlcorresponding to a driver's emotional state in a control system using anin-vehicle gesture input according to another embodiment of the presentinvention.

FIG. 7 is an exemplary diagram showing an object selection gesture in acontrol system using an in-vehicle gesture input according to anotherembodiment of the present invention.

FIG. 8 is an exemplary diagram of a control corresponding to an objectselection gesture and voice input in a control system using anin-vehicle gesture input according to another embodiment of the presentinvention.

FIGS. 9 and 10 are exemplary diagrams of a control corresponding to adevice selection gesture and voice input in a control system using anin-vehicle gesture input according to another embodiment of the presentinvention.

FIG. 11 is an exemplary diagram showing a gesture input for a pedestrianguidance indication in a control system using an in-vehicle gestureinput according to another embodiment of the present invention.

FIG. 12 is an exemplary diagram showing a phrase indication for apedestrian in a control system using an in-vehicle gesture inputaccording to another embodiment of the present invention.

FIG. 13 is an exemplary diagram showing a warning indication for anearby vehicle in a control system using an in-vehicle gesture inputaccording to another embodiment of the present invention.

FIG. 14 is an exemplary diagram of a control corresponding to a gestureinput upon detection of no pedestrian in a control system using anin-vehicle gesture input according to another embodiment of the presentinvention.

FIG. 15 is an exemplary diagram showing a gesture input for externalcommunication to a nearby vehicle in a control system using anin-vehicle gesture input according to another embodiment of the presentinvention.

FIG. 16 is an exemplary diagram showing a phrase indication as an aspectof external communication to a nearby vehicle in a control system usingan in-vehicle gesture input according to another embodiment of thepresent invention.

FIG. 17 is an exemplary diagram showing a driver's selection gesture andvoice input in a situation where images are layered in a windshield in acontrol system using an in-vehicle gesture input according to anotherembodiment of the present invention.

FIG. 18 is an exemplary diagram showing information display and imagemovement corresponding to a driver's selection gesture and voice inputin a control system using an in-vehicle gesture input according toanother embodiment of the present invention.

FIG. 19 is an exemplary diagram showing an image area return in acontrol system using an in-vehicle gesture input according to anotherembodiment of the present invention.

DETAILED DESCRIPTION

These and other objects, advantages and features of the presentinvention, and implementation methods thereof will be clarified throughthe following embodiments described with reference to the accompanyingdrawings.

The present invention may, however, be embodied in different forms andshould not be construed as limited to the embodiments set forth herein.Rather, these embodiments are provided so that this disclosure willfully convey the objects, configurations, and effects of the presentinvention to those skilled in the art. The scope of the presentinvention is defined solely by the appended claims.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a,” “an,” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. The terms “comprises” and/or “comprising,” when used in thisspecification, specify the presence of stated elements, steps,operations, and/or components, but do not preclude the presence oraddition of one or more other elements, steps, operations, and/orcomponents.

FIG. 1 is a block diagram showing a control system using an in-vehiclegesture input according to an embodiment of the present invention.

A control system 100 using an in-vehicle gesture input according to anembodiment of the present invention includes an input unit 110configured to receive a user's gesture, a memory 120 configured to storea control program using an in-vehicle gesture input therein, and aprocessor 130 configured to execute the control program. The processor130 transmits a command to perform different functions on the samegesture of a user according to the usage pattern.

The input unit 110 according to an embodiment of the present inventionreceives information obtained by a 3D camera (e.g., ToF, Structure,Stereo, and DSM (Driver Status Monitoring) cameras) detecting a gestureaction of a user (a driver or a fellow passenger) and receives anoperation state (usage pattern) involved in the user's gesture.

The processor 130 according to an embodiment of the present inventiontransmits a command for executing different in-vehicle functions for thesame gesture according to the usage pattern including the operationstate involved in the user's gesture.

Contact with a touchpad 300 or an armrest is an example of the operationstate involved in the user's gesture.

FIG. 2 is an exemplary diagram of a contact swipe gesture input in acontrol system using an in-vehicle gesture input according to anembodiment of the present invention, and FIG. 3 is an exemplary diagramof a non-contact swipe gesture input in a control system using anin-vehicle gesture input according to an embodiment of the presentinvention.

In this case, as shown in FIGS. 2 and 3 , it is assumed that a swipegesture input is a gesture of a user swiping right with the right hand(t1→t2).

Referring to FIG. 2 , a windshield area includes a first area A1 and asecond area A2.

As shown in FIG. 2 , when a user inputs a gesture of swiping right withthe right hand while the arm is in contact with the touchpad 300, theinput unit 110 receives information regarding a user gesture action,which is acquired by the camera 200, and receives information regardingan arm mounting state, which is acquired by the touchpad 300.

The processor 130 transmits a command for moving an image display areafrom the first area to the second area (A1→A2), and the command is anaction command corresponding to the gesture and arm mounting state.

For example, as shown in FIG. 2 , when a preset swipe gesture is inputwhile the arm is in contact with the touchpad 300, a navigation screenmay be moved from the first area A1 to the second area A2.

Referring to FIG. 3 , when a user inputs a gesture of swiping right withthe right hand (t1→t2) while the arm is not in contact with the touchpad300, the input unit 110 receives information regarding a user gestureaction, which is acquired by the camera 200, and receives informationregarding an arm mounting state, which is acquired by the touchpad 300.

Through the corresponding gesture, the processor 130 transmits a“Rewind” command for image content currently being displayed in thefirst area A1.

For example, movie content is being displayed in the first area A1. Theprocessor 130 transmits a “Rewind” command (

), which is a function corresponding to the swipe gesture, among Play (

), Pause (

), Rewind (

), Fast-forward (

), and Stop (▮) of the movie content and changes a play start point ofthe corresponding content (e.g., the play start point is changed from1:00:04 at present to 0:59:54 which is 10 seconds ahead according to theswipe action).

According to an embodiment of the present invention, the function“Rewind” may be provided differently in consideration of a distancebetween a start point and an end point of a driver's swipe action (e.g.,go back 10 seconds when the swipe action is shortened to 10 cm, and goback 30 seconds when the swipe action is lengthened to 30 cm).

That is, according to an embodiment of the present invention, differentfunctions may be provided (a function of moving an area for displayingan image is provided when contact is made with a touchpad, and afunction of moving a play start point of image content when contact isnot made with a touchpad) for the same gesture (a gesture of swipingright with the right hand) according to an operation state (e.g.,whether contact is made with a touchpad) involved in a user gesture.

The processor 130 according to an embodiment of the present inventiondifferently performs a control corresponding to a gesture inputaccording to a user's hand position and arm mounting form. For example,the processor 130 may give priority to a touch when a hand is in contactwith a touchpad and may perform a manipulation corresponding to agesture when a hand is away from a touchpad.

According to an embodiment of the present invention, a gesturerecognition concept changes depending on whether contact is made with atouchpad or an armrest. For example, when a gesture is input to an IVIarea while the elbow leans against a console touchpad or an armrest, theprocessor 130 recognizes an air gesture for a designated hand action andtransmits a corresponding system operation command.

In this case, the system operation command includes switching an IVIdisplay area screen, selecting a mode, executing an auxiliary functionof other vehicles, etc.

According to an embodiment of the present invention, when a contactstate is changed to a non-contact state, a user interface (UI) changes,and a manipulation gesture corresponding to a virtual touch (an airtouch) is recognized.

A user may perform a virtual touch manipulation in a non-contact stateby raising his or her arm to manipulate a windshield (a head-up display,a large screen display, etc.) or other items and may input a quickexecution request through a virtual touch operation according to thechange in the UI (the image included in the windshield or the IVIdisplay).

FIG. 4 is a block diagram showing a control method using an in-vehiclegesture input according to an embodiment of the present invention.

The control method using an in-vehicle gesture input according to anembodiment of the present invention includes receiving operation stateinformation and a gesture (S410), determining whether the operationstate information satisfies a preset condition (S420) and providing afirst function or a second function corresponding to a first gestureaccording to whether the operation state information satisfies thepreset condition (S430 and S440).

In S410, an occupant's gesture (e.g., an action of swiping right with ahand) is received from an image acquisition unit which recognizes anoccupant's gesture in a vehicle, and also operation state information isreceived.

As an example, in S410, whether a user's arm is in contact with atouchpad is received as the operation state information.

In S420, it is determined whether the user's arm is in contact with thetouchpad. In S430, a first function (a function of moving an area inwhich content is displayed) corresponding to the gesture (a firstgesture) received in S410 is provided in S430 when the user's arm is incontact with the touchpad.

In S440, a second function (a function of adjusting or moving back aplay start time of image content being displayed) corresponding to thegesture (the first gesture) received in S410 is provided when it isdetermined that the user's arm is not in contact with the touchpad inS420.

According to an embodiment of the present invention, it is possible toincrease user convenience not only by recognizing a gesture but also byexecuting different functions for the same gesture depending on thesituation to perform matching between gestures and functions inconsideration of operation state information regarding the gestures.

As another example, in S410, a driver's emotional state, a driver'svoice input state, a driver's drowsiness state, and the like arereceived as the operation state information. Accordingly, it is possibleto provide different functions for the same gesture.

For example, in S410, it is assumed that a driver's emotional state isreceived as the operation state information and that music is beingstreamed.

In this case, the driver's emotional state is determined by analyzingused to analyze a face image of the driver acquired through anin-vehicle camera to determine whether the driver is calm or frowning(angry).

Alternatively, biometric information is shared from the driver'swearable device, and thus it is possible to check the driver's currentemotional state.

Alternatively, through a schedule application of the driver'ssmartphone, it is possible to determine whether the driver is onvacation and thus feels calm or is on a tight schedule and thus feelstired.

When the face image of the driver acquired through the in-vehicle camerais determined as indicating that the driver is frowning and the driverinputs a gesture (an action of swiping right with the right hand)corresponding to a basic function of playing the next song in aplaylist, in S420, it is determined that the driver's emotional stateindicates that a preset condition corresponding to “angry” is satisfied,and in S430, a first function of recommending and streaming a song thatmakes the driver feel better as the next song is provided instead ofjust playing the next song in the playlist in response to thecorresponding gesture.

In this case, in S430, a song to which a driver has listened and forwhich good feedback has been provided is recommended and streamed.Alternatively, a song loved by multiple different users is recommendedand streamed through big data analysis. Alternatively, a song that thedriver will like is recommended and streamed in consideration of season,weather, the driver's gender and age, the driver's favorite era, singer,and genre.

When the face image of the driver acquired through the in-vehicle camerais determined as indicating that the driver has a calm expression andthe driver inputs a gesture of playing the next song (an action ofswiping right with the right hand), in S420, it is determined that thedriver's emotional state is calm such that a preset condition is notsatisfied, and in S440, a second function (a basic function) for playingthe next song in the playlist is provided on the basis of thecorresponding gesture.

As another example, in S410, the driver's voice is received, and agesture of raising the right hand up is received.

When it is determined in S420 that the driver says “temperature,” afirst function of determining that the driver's voice input controls anair conditioner and raising the set temperature by one degree isprovided in S430.

When it is determined in S420 that the driver says “passenger seatwindow,” a second function of determining that the driver's voice inputcontrols opening or closing of a passenger seat window and raising andclosing the passenger seat window is provided in S430.

As described above, according to an embodiment of the present invention,it is possible to provide different functions for the same gestureaccording to the operation state information involved in the usergesture.

FIG. 5 is a block diagram of a control system using an in-vehiclegesture input according to another embodiment of the present invention,and FIG. 6 is an exemplary diagram showing a gesture input controlcorresponding to a driver's emotional state in a control system using anin-vehicle gesture input according to another embodiment of the presentinvention.

A control system using an in-vehicle gesture input according to anotherembodiment of the present invention includes a driver state recognitionunit 110 a configured to recognize a driver's state, a gesture inputunit 110 b configured to receive gesture information, and a control unit140 configured to selectively match and provide different functions toand for the same gesture information according to the driver's state andthe gesture information.

The driver state recognition unit 110 a is configured to receive adriver's emotion state information and may generate emotion stateinformation from a driver's facial expression through an in-vehicle DSMcamera, generate emotion state information from a driver's biometricinformation through the driver's wearable device, or generate a driver'spsychological state information in connection with a scheduleapplication of the driver's smart device.

An embodiment in which different functions are selectively matched toand provided for the same gesture information according to the driver'semotion state information and gesture information will be describedbelow.

As shown in FIG. 6 , when a gesture of swiping right with the right hand(t1→t2) is input while a driver is driving and listening to music, thegesture input unit 110 b receives the driver's gesture informationrecognized through the camera 200.

A basic function connected with the gesture of swiping right with theright hand (t1→t2) is set to a function of playing the next song.

In this case, when the driver state recognition unit 110 a classifiesthe driver's facial expression received through the camera 200 as“angry/agitated,” the control unit 140 controls execution of a separatefunction to recommend and play a song that improve the driver's mood,rather than simply playing the next song in the playlist.

For example, the control unit 140 recommends and plays a songrecommended as music that makes multiple users feel good, acorresponding user's favorite song (a user's favorite song having ahistory built as being mainly played based on season, weather, and timeperiod), a song determined as having a change that makes a driver feelbetter through an analysis of driver data, and the like.

On the other hand, when the driver state recognition unit 110 aclassifies the driver's facial expression recognized through the camera200 as “calm,” the control unit 140 performs a control to play the nextsong in the playlist according to the basic function of thecorresponding gesture.

According to this embodiment, it is possible to operate differentfunctions for the same gesture depending on the driver's emotionalstate, which preferably corresponds to an entertainment function whichis manipulated due to the gesture.

When it is determined through the driver state recognition unit 110 aand the gesture input unit 110 b that the driver's pleasant facialexpression and the driver's hand or face motion are present, the controlunit 140 performs a control to promote the driver's good mood through amood lamp equalizing effect.

When the driver state recognition unit 110 a recognizes the driver'sfrowning expression and the gesture input unit 110 b recognizes agesture of the driver blocking his or her nose, the control unit 140performs a control to switch between outdoor and indoor units of an airconditioner or to operate an air cleaning mode.

A swipe action that is input through the gesture input unit 110 b isbasically set as a gesture for performing an action of changing a screenor content which is being displayed. When a call comes to the driver'ssmartphone connected with an in-vehicle system and a frowning face or anegative voice is recognized through the driver state recognition unit110 a, the control unit 140 transmits an incoming call rejection commandaccording to the corresponding swipe action.

When the gesture input through the gesture input unit 110 b is a gestureof the driver continuously touching or hitting his or her chest and thedriver's frowning expression is recognized from the driver staterecognition unit 110 a, the control unit 140 performs a control todetermine that the driver has an abnormal health condition and to informthe driver of a nearby hospital location.

Also, the control unit 140 transmits, to nearby rescue workers,hospitals, and the driver's acquaintances, and the like, a notificationabout the driver's abnormal health condition.

According to the aforementioned embodiment, even when a separatebiometric sensor is not provided, it is possible to determine whetherthere is an abnormality in the driver's health state according to thedriver's face recognition result and behavior analysis and to providecorresponding vehicle functions (e.g., hospital location guidance andrescue worker and acquaintance contact).

The control unit 140 switches a vehicle driving mode according to thestate recognition result of the driver state recognition unit 110 a.

The control unit 140 operates the vehicle in an autonomous driving modewhen the driver's emotional state is classified as “angry/agitated.” Thecontrol unit 140 automatically receives a selection of a mode forautonomous driving or manual driving from the driver or automaticallyexecutes a recently stored driving mode when the driver's emotionalstate is classified as “calm.”

In this case, the driver state recognition unit 110 a may receiveemotion information from the driver's mobile terminal before the drivergets in the vehicle and may recognize the driver's emotional statethrough the above camera 200 or the like after the driver gets in thevehicle.

In addition, the driver state recognition unit 110 a may not onlyrecognize the driver's state from the driver's actions or facialexpressions but may estimate the driver's state through externalinformation such as a driving pattern during manual driving.

For example, when the driver is making a sudden departure, sudden stop,or sudden lane change, it is possible to estimate that the driver isdriving in an “angry/agitated” state and to automatically change adriving mode or recommend a driving mode change according to a result ofthe estimation.

As the driving proceeds after the driver gets in the vehicle, the driverstate recognition unit 110 a monitors the emotional state, and thecontrol unit 140 maintains the aforementioned autonomous driving mode orswitches the manual driving mode to the autonomous driving mode inconsideration of a result of the monitoring.

The control unit 140 performs a control to display the driver'semotional state to nearby vehicles.

In this case, a phrase may be displayed through an externalcommunication display, or the driver's emotion state information istransmitted through V2V.

For example, when a driver of a vehicle “A” is manually driving in an“angry/agitated” state, the driving may be violent. In this case, thevehicle “A” may inform nearby vehicles of the driver's emotional state,and thus a nearby vehicle “B” (for an autonomous driving mode) or adriver of the vehicle “B” (for a manual driving mode) can prepare forthe violent driving of the vehicle “A” (e.g., securing a safe distanceand refraining from changing lanes ahead of the vehicle “A”).

As another embodiment, the driver state recognition unit 110 a receivesstate information regarding whether the driver is drowsy while driving.

When the state information indicates that the driver's state correspondsto the drowsy driving, such as the driver's eyes closing or headnodding, is recognized through the camera 200 or that the driver inputsa gesture of rubbing his or her eyes or squeezing the back of his or herneck through the gesture input unit 110 b, the control unit 140determines that the driver is drowsy while driving.

When a gesture of swiping right with the right hand (t1→t2) is inputwhile the driver is driving and listening to music, the gesture inputunit 110 b receives the driver's gesture information recognized throughthe camera 200.

The gesture of swiping right with the right hand (t1→t2) is a gesture toplay the next song. In this case, when the driver's drowsy driving ispredicted through the driver state recognition unit 110 a and thegesture input unit 110 b, the control unit 140 may perform a control torecommend and play a song with a fast tempo that can remove the driver'sdrowsiness rather than simply playing the next song in the playlist.

Alternatively, according to the corresponding gesture, the control unit140 plays the next song in the playlist in consideration of the driver'sintention and increases the volume by 1.5 times in consideration of thedriver's drowsy driving.

FIG. 7 is an exemplary diagram showing an object selection gesture in acontrol system using an in-vehicle gesture input according to anotherembodiment of the present invention.

The control system using an in-vehicle gesture input according toanother embodiment of the present invention includes an input unitconfigured to receive a user's gesture and voice, a memory configured tostore a control program using an in-vehicle gesture input therein, and aprocessor configured to execute the control program. The processortransmits a device control command according to the user's gesture andvoice.

Referring to FIG. 7 , the driver inputs a gesture of selecting an objectof interest near the vehicle using a hand H.

For example, another vehicle C, pedestrian P, traffic sign, building,and the like are present near the vehicle. The driver may conduct agesture of selecting an external object using the hand H and also maytrack the driver's gaze information to select an external object.

According to this embodiment, the processor performs a control todisplay information (speed, vehicle type, vehicle number, and othervehicle sharing message information) regarding a selected front vehicleC according to a virtual gesture selection using the driver's hand, andthe front vehicle information is displayed in various areas such as anaudio, video, and navigation (AVN) screen, a cluster gauge, a head-updisplay (HUD), and a windshield glass.

The processor performs a control to display information (motion anddistance from road) of a selected pedestrian, information (name, enteredshops, and facilities) of a selected building, and information of aselected traffic sign according to a virtual gesture selection using thedriver's hand, and the pieces of information are displayed in variousareas as described above.

The input unit receives the driver's vocal utterance as well as thedriver's gesture, and the processor transmits a message to a selectedobject (thing, vehicle, etc.) according to the voice reception.

For example, it will be described with reference to FIG. 8 that ataillight of the front vehicle C is broken.

The input unit receives object selection information indicating that thefront vehicle C has been selected with the driver's hand H and receivesvoice information indicating that “the taillight is broken,” which isthe driver's voice.

The processor transmits a notification about a taillight failure to thefront vehicle C, which is the corresponding object, according to thedriver's gesture and voice, and the notification is provided through thewindshield or AVN of the front vehicle through V2V.

Alternatively, the above-mentioned notification may be recognized by adriver of a corresponding vehicle C through an external communicationdevice (e.g., a display for displaying text) of the vehicle. To thisend, a text message indicating a taillight failure is displayed throughan external communication device disposed on the rear side of thevehicle when the corresponding vehicle overtakes the vehicle, and a textmessage indicating a taillight failure is displayed through an externalcommunication device disposed on a lateral side of the vehicle toward acorresponding vehicle when the corresponding vehicle is travelingalongside the vehicle.

For example, when a vehicle C is traveling without turning on aheadlight in a night driving situation and a voice indicating “Turn on alamp and drive safely” is input together with a gesture of selecting thecorresponding vehicle C through the input unit, a text messageindicating the voice is displayed through the above-described externalcommunication device, or a notification is provided to the front vehiclethrough V2V.

Also, when the corresponding front vehicle C is swaying from side toside and drowsy driving is suspected, the input unit receives gestureinformation and receives voice information indicating “Be careful” byselecting the corresponding vehicle C with the driver's hand H.

In this case, the processor urgently transmits a message which is basedon the voice information of the driver and which indicates not to drivedrowsy to the front vehicle through V2V, and the message causes thefront vehicle that has received the message to generate seat vibrationand thus to provide an alarm to the driver of the front vehicle.

Also, as described above, the processor performs a control to display,as a text message, a notification message indicating not to drive drowsyto the corresponding vehicle C through an external communication deviceaccording to the driver's gesture and voice.

When the driver's voice indicating “Please make a reservation for XXpeople at OO o'clock” is input through the input unit along with agesture of selecting a building near the vehicle, the processor deliversa reservation message to the building.

When the gesture of selecting the building near the vehicle is inputthrough the input unit, the processor displays phone number informationof the corresponding building. When a request for a call connection isreceived by the driver's voice or gesture, a call is originated to acorresponding phone number.

When the driver's voice indicating “Traffic accident, please call 119”is received along with a gesture of selecting a traffic sign through theinput unit, the processor transmits an accident report message to 119Control Center. In this case, by transmitting an accident report messageincluding information (location information) of the traffic sign, theprocessor delivers information regarding safe and emergent situationsaccording to information regarding the gesture of the driver selectingthe traffic sign.

When a pedestrian P who is jaywalking near the vehicle is selectedthrough the input unit and the driver's voice indicating “Don't jaywalk”is also input, the processor displays a warning phrase as a text messageto the corresponding pedestrian through an external communicationdevice.

The processor identifies a speaker, which is one of a driver and apassenger, and classifies the importance of a transmission messagethrough the storage and determination of voice information.

The importance and the alarm control vary depending on whether thespeaker is a driver or a passenger (passenger content filtering), andauthority related to the importance is transferred from the driver tothe passenger to allow delivery of messages.

Also, the processor delivers a common shared message through a voice sothat information such as an in-front road situation can be checked byother vehicles.

FIGS. 9 and 10 are exemplary diagrams of a control corresponding to adevice selection gesture and voice input in a control system using anin-vehicle gesture input according to another embodiment of the presentinvention.

The processor performs a control to basically provide an air gesture orair touch function according to the driver's gesture received throughthe input unit.

This relates to basic gesture actions such as button clicking and modeexecution and includes selection and execution of buttons and modeswhich are represented in the screen.

The input unit receives information regarding an in-vehicle deviceindicated by the user's gesture and also receives the user's voiceinformation.

The processor executes an operation corresponding to the voiceinformation (a short-cut function and a quick execution function) inconsideration of the user's voice information and the informationregarding the in-vehicle device selected according to the gesturereceived through the input unit.

As shown in FIG. 9 , when the input unit receives a selection of thearea of the air conditioner 600 according to a gesture of a user's handH (t1→t2) and receives the user's voice (e.g., strong or weak withrespect to air blowing; up, down, left, or right with respect to the airdirection, an air conditioner, temperature, and the like with respect toan air conditioning mode, etc.), the processor controls operationexecution for an air conditioner, which is an in-vehicle devicecorresponding to the area indicated by the user's hand.

As shown in FIG. 10 , when the input unit receives a selection of thearea of the navigation 400 according to a gesture of a user's hand H(t1→t2) and receives the user's voice (e.g., a destination), theprocessor controls operation execution, such as for moving to adestination search screen, for displaying a destination search list, orfor automatically setting a destination.

When the input unit receives a selection of the area of an audio deviceaccording to a gesture of a user's hand H (t1→t2) and receives theuser's voice (e.g., music file title, singer, genre, etc.), theprocessor controls operation execution, such as for moving to a musicfile search screen, for displaying a music file search list, or forautomatically playing music according to the user's voice.

The processor controls the execution of devices in consideration of thedriver's gesture and the driver's voice, and thus it is possible todifferently control function execution even for the same gestureaccording to different voice inputs of the driver.

For example, when the driver says “volume” and performs a gesture ofraising a hand up, the processor controls execution to increase thevolume of the audio device. When the driver says “air blowing” andperforms a gesture of raising a hand up, the processor controlsexecution to increase the air volume of the air conditioner.

Also, when the driver says “screen” and performs a gesture of raising ahand up, the processor controls execution to increase the brightness ofthe screen.

Also, when the driver says “navigation” and performs a gesture ofraising a hand up, the processor enlarges a map screen.

When the driver's gesture is to touch his or her lips with the indexfinger and the driver's voice is “Hush,” the processor mutes sound thatis emitted from the AVN.

Also, when main functions are simultaneously performed, a short cutfunction may be provided through a specific gesture. When a call isincoming while the sound is being emitted from the AVN (listening toradio, music, etc.), the incoming call may be rejected through thegesture.

Also, when the driver says “music” and inputs a specific gesture, theprocessor searches (drags) a content/menu/play list according to thegesture. When the driver says “OK, stop,” the processor stops aselection window in a position or range of a corresponding music file.

Also, when the driver makes a gesture of raising his or her thumb upalong with positive language (e.g., good, best, etc.) while listening tomusic, the processor may perform a procedure including storinginformation about the music, downloading a music file, adding the musicto the playlist, and transmitting positive feedback through in-vehicleSNS.

The processor transmits an image (e.g., entertainment in a windshielddisplay) selected through a gesture and other input information (atouch, a voice, a surrounding environment, etc.) to other devices (amobile terminal, a backseat passenger display, other vehicles).

In this case, when the image is transmitted to the backseat passengerdisplay, the screen is transmitted without modification. When the imageis transmitted to the mobile terminal, essential information isextracted in consideration of a screen size, and a modified image istransmitted.

When the image is transmitted to other vehicles, the transmission isperformed through V2X communication and is executed in consideration ofcongestion control in traffic congestion areas.

In this case, when a packet loss rate is high, the image is nottransmitted and only safety-related information is transmitted.

The processor transmits only a gesture symbol without transmitting imageinformation itself, and a device which has received the gesture symbolprocesses a subsequent function control corresponding to the gesturesymbol. Thus, it is possible to reduce the amount of data packets.

FIG. 11 is an exemplary diagram showing a gesture input for a pedestrianguidance indication in a control system using an in-vehicle gestureinput according to another embodiment of the present invention.

The control system using an in-vehicle gesture input according toanother embodiment of the present invention includes an input unitconfigured to receive surrounding situation information and a user'sgesture, a memory configured to store a control program using anin-vehicle gesture input therein, and a processor configured to executethe control program. The processor transmits a command to executedifferent functions for the same gesture of the user according to thesurrounding situation information and the user's gesture.

In this embodiment, information regarding whether there is a pedestrianwho tries to cross a road at a crosswalk in front of the vehicle is usedas the surrounding situation information.

Referring to FIG. 11 , there is a pedestrian P who is waiting to crossat a crosswalk in front of the vehicle (at a crosswalk with no trafficlights), and the driver inputs a gesture of waving a hand (t1→t2→t3→t4)to instruct the pedestrian to cross.

Conventionally, pedestrians cross after perceiving that a vehicle willstop by looking at the driver's hand signal. Such identification isdifficult when the inside of the vehicle is not visible because of atinting or a night driving situation. Therefore, in this embodiment, theprocessor performs a control to display a message for crossing a road toa pedestrian through the driver's gesture recognized through thein-vehicle camera 200.

Referring to FIG. 12 , when a user's gesture as shown in FIG. 11 isreceived in the above-described situation, an external communicationdevice 700 disposed on the front side of the vehicle displays a phrase“Cross” and enables the pedestrian P to check the phrase and then cross.

Such pedestrian guidance may be provided through an external directionalspeaker or the like in the form of an audio message as well as theabove-described text message. Further, it is also possible to inducepedestrians to cross by illuminating a crosswalk in a green colorthrough an external lighting device.

When there is no pedestrian ahead, the corresponding gesture of thedriver is input to perform an original command (a command of movingcontent located in a first area to a second area) as shown in FIG. 14 ,and thus the area of the content is moved.

That is, according to this embodiment, it is possible to executedifferent functions for the same gesture according to the surroundingsituation information (information regarding whether there is apedestrian who is waiting to cross).

FIG. 13 is an exemplary diagram showing a warning indication for anearby vehicle in a control system using an in-vehicle gesture inputaccording to another embodiment of the present invention.

When a pedestrian P is crossing a road at a crosswalk in front of thevehicle and information regarding the presence of a vehicle Capproaching from a rear side is received, the processor provides awarning notification to the vehicle C because the pedestrian is notvisible due to the vehicle and thus there is a risk of collision withthe pedestrian caused by the vehicle C.

The processor controls a lamp of the vehicle to blink in a direction inwhich the vehicle C is approaching and induces the deceleration of thevehicle C.

Alternatively, the processor displays a message “Beware of pedestrianahead” through an external communication device 700 b disposed on therear side of the vehicle and induces the vehicle C to recognizeinformation regarding the presence of a pedestrian ahead and thendecelerate.

Also, the process switches a phrase of an external communication device700 a disposed on the front side of the vehicle to display a message“Beware of approaching vehicle,” and thus induces the pedestrian P toperceive whether the vehicle C is approaching and then safely cross.

FIG. 15 is an exemplary diagram showing a gesture input for externalcommunication to a nearby vehicle in a control system using anin-vehicle gesture input according to another embodiment of the presentinvention.

For example, when a user performs a gesture of lifting a hand H up tothe left or right, the processor determines that the driver intends toobtain an understanding of a lane change (merging) and displays amessage through an external communication device.

For example, as shown in FIG. 16 , according to a gesture direction,turn signal lamps operate simultaneously, and the processor displays aphrase “Thank you” to obtain an understanding through an externalcommunication device 700 c disposed on a side of the vehicle.

In this case, it is preferable that the driver's gesture be recognizedwhen the gesture is performed at a predetermined height or more from aseating posture.

When a lane change is completed, the processor displays a phrase “Sorry”to obtain an understanding through the external communication devicedisposed on the rear side of the vehicle.

Also, when the vehicle suddenly stops, the processor may display aphrase “Sorry” to obtain an understanding through the externalcommunication device disposed on the rear side of the vehicle accordingto a gesture of the driver lifting a hand.

The processor provides different functions for the same gesture inconsideration of the driver's gesture and driving history information.

For example, the processor provides a function of increasing the airblowing of the air conditioner (a first function) according to thegesture of the driver lifting a hand in the case of a generaldeceleration situation, not an emergency stop situation, and provides afunction of displaying a phrase “Sorry” to obtain an understanding (asecond function) according to the gesture of the driver lifting a handin an emergency stop situation.

FIG. 17 is an exemplary diagram showing a driver's selection gesture andvoice input in a situation where images are layered in a windshield in acontrol system using an in-vehicle gesture input according to anotherembodiment of the present invention.

Referring to FIG. 17 , the camera 200 delivers, to the input unit,gesture information input through a driver's hand H.

It is possible to determine what the driver's object (building or thing)of interest is by recognizing the driver's gaze direction in addition tothe driver's hand H and performing eye-tracking.

FIG. 17 shows that navigation information (layer 2; navigation,entertainment, etc.) is displayed in a preset area A1 of an in-vehiclewindshield (layer 1; main screen) and that a driver wants to checkinformation regarding an outdoor object hidden by displaying navigationinformation while staring at the area A1.

That is, the driver inputs a gesture of selecting an object with his orher hand H and makes an utterance such as “Where is it?” and “Checkinformation” through voice V.

Referring to FIG. 18 , the processor displays information regarding thedriver's object of interest on the windshield according to the driver'sinput gesture and voice, as shown in FIG. 17 .

For example, when the object is a sculpture for the Big-O Show locatedin Yeosu, South Korea, the performance information of the Big-O Show(start time: every 30 minutes from 18:00 to 22:00, time required: 15minutes) may be displayed in a predetermined area R1 of the windshieldand may be provided through voice as well as through text.

Also, in conjunction with the provision of such information, asubsequent function related to a reservation for a performance may beprovided.

The above-described example is a description of a sculpture for aperformance, and when the driver's object of interest is a restaurant,menu information, operation information, etc. are displayed, and asubsequent function such as the transmission of a reservation messagefor a restaurant reservation or the connection of a call to therestaurant is provided.

When the information regarding the driver's object of interest isdisplayed, navigation screen information that has been displayed in thearea A1 is delivered to the AVN 400 of the vehicle and is displayed bythe AVN 400.

When there are three or more layers and the above-described gestureinput is made from the driver, it is possible to move and display eachimage to and on a plurality of displays (windshield, AVN, cluster, CMSmonitor, and the like).

The processor performs a screen movement control to perform image layerprocessing and performs a control to display information to bepreferentially shown according to a user's intention (informationregarding a sculpture R) on a main screen (the windshield at which thedriver is gazing) and to move an image which has been displayed on themain screen to a subsequent display (AVN, cluster, passenger seatdisplay, etc.).

The processor may adjust the rankings of subsequent displays accordingto a vehicle driving mode (manual or autonomous driving).

In this case, by operations of the driver's gesture, voice, eyetracking, etc., a displayed screen may be moved to another display todisplay information, and the display may be automatically moved after aguidance voice indicating “Move to OO display” is provided.

FIG. 19 is an exemplary diagram showing an image area return in acontrol system using an in-vehicle gesture input according to anotherembodiment of the present invention.

When the driver checks the above-described information regarding thesculpture and then makes an utterance such as “OK” or stops displayingthe information regarding the sculpture through a swipe action or when apredetermined time (e.g., 20 seconds) has passed after the informationregarding the sculpture is displayed, the navigation screen that hasbeen moved to the AVN 400 is returned and moved back to the area A1 ofthe windshield where the navigation screen was displayed.

In this case, when the moved content is not real-time information suchas a navigation screen (e.g., movie content), the movie content istemporarily paused at the time of movement as shown in FIG. 18 andreplayed at the time of returning to the original area as shown in FIG.19 .

Meanwhile, the control method using an in-vehicle gesture inputaccording to an embodiment of the present invention may be implementedin a computer system or recorded on a recording medium. The computersystem may include at least one processor, memory, user input device,data communication bus, user output device, and storage. Theabove-described elements perform data communication through the datacommunication bus.

The computer system may further include a network interface coupled to anetwork. The processor may be a central processing unit (CPU) or asemiconductor device for processing instructions stored in a memoryand/or a storage.

The memory and storage may include various types of volatile ornon-volatile storage media. For example, the memory may include aread-only memory (ROM) and a random access memory (RAM).

Accordingly, the control method using an in-vehicle gesture inputaccording to an embodiment of the present invention may be implementedas a computer-executable method. When the control method using anin-vehicle gesture input according to an embodiment of the presentinvention is performed by a computer device, computer-readableinstructions may implement the control method according to the presentinvention.

Meanwhile, the control method using an in-vehicle gesture inputaccording to an embodiment of the present invention may be embodied ascomputer-readable codes on a computer-readable recording medium. Thecomputer-readable recording medium includes any type of recording mediumin which data that can be decrypted by a computer system is stored. Forexample, the computer-readable recording medium may include a ROM, aRAM, a magnetic tape, a magnetic disk, a flash memory, an optical datastorage device, and the like. Further, the computer-readable recordingmedia can be stored and carried out as codes that are distributed in acomputer system connected to a computer network and that are readable ina distributed manner.

According to an embodiment of the present invention, by performing acontrol to identify a gesture control input for each situation andexecuting a different function for each situation even when the samegesture is made to reduce the number of gestures to be learned by adriver to execute functions, it is possible to increase userconvenience, and it is also possible for a driver to execute a desiredfunction through an intuitive gesture input.

Advantageous effects of the present invention are not limited to theaforementioned effects, and other effects not described herein will beclearly understood by those skilled in the art from the abovedescription.

The present invention has been described above with respect toembodiments thereof. Those skilled in the art should understand thatvarious changes in form and details may be made therein withoutdeparting from the essential characteristics of the present invention.Therefore, the embodiments described herein should be considered from anillustrative aspect rather than from a restrictive aspect. The scope ofthe present invention should be defined not by the detailed descriptionbut by the appended claims, and all differences falling within a scopeequivalent to the claims should be construed as being encompassed by thepresent invention.

The components described in the example embodiments may be implementedby hardware components including, for example, at least one digitalsignal processor (DSP), a processor, a controller, anapplication-specific integrated circuit (ASIC), a programmable logicelement, such as an FPGA, other electronic devices, or combinationsthereof. At least some of the functions or the processes described inthe example embodiments may be implemented by software, and the softwaremay be recorded on a recording medium. The components, the functions,and the processes described in the example embodiments may beimplemented by a combination of hardware and software.

The method according to example embodiments may be embodied as a programthat is executable by a computer, and may be implemented as variousrecording media such as a magnetic storage medium, an optical readingmedium, and a digital storage medium.

Various techniques described herein may be implemented as digitalelectronic circuitry, or as computer hardware, firmware, software, orcombinations thereof. The techniques may be implemented as a computerprogram product, i.e., a computer program tangibly embodied in aninformation carrier, e.g., in a machine-readable storage device (forexample, a computer-readable medium) or in a propagated signal forprocessing by, or to control an operation of a data processingapparatus, e.g., a programmable processor, a computer, or multiplecomputers. A computer program(s) may be written in any form of aprogramming language, including compiled or interpreted languages andmay be deployed in any form including a stand-alone program or a module,a component, a subroutine, or other units suitable for use in acomputing environment. A computer program may be deployed to be executedon one computer or on multiple computers at one site or distributedacross multiple sites and interconnected by a communication network.

Processors suitable for execution of a computer program include, by wayof example, both general and special purpose microprocessors, and anyone or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. Elements of a computer may include atleast one processor to execute instructions and one or more memorydevices to store instructions and data. Generally, a computer will alsoinclude or be coupled to receive data from, transfer data to, or performboth on one or more mass storage devices to store data, e.g., magnetic,magneto-optical disks, or optical disks. Examples of informationcarriers suitable for embodying computer program instructions and datainclude semiconductor memory devices, for example, magnetic media suchas a hard disk, a floppy disk, and a magnetic tape, optical media suchas a compact disk read only memory (CD-ROM), a digital video disk (DVD),etc. and magneto-optical media such as a floptical disk, and a read onlymemory (ROM), a random access memory (RAM), a flash memory, an erasableprogrammable ROM (EPROM), and an electrically erasable programmable ROM(EEPROM) and any other known computer readable medium. A processor and amemory may be supplemented by, or integrated into, a special purposelogic circuit.

The processor may run an operating system (OS) and one or more softwareapplications that run on the OS. The processor device also may access,store, manipulate, process, and create data in response to execution ofthe software. For purpose of simplicity, the description of a processordevice is used as singular; however, one skilled in the art will beappreciated that a processor device may include multiple processingelements and/or multiple types of processing elements. For example, aprocessor device may include multiple processors or a processor and acontroller. In addition, different processing configurations arepossible, such as parallel processors.

Also, non-transitory computer-readable media may be any available mediathat may be accessed by a computer, and may include both computerstorage media and transmission media.

The present specification includes details of a number of specificimplements, but it should be understood that the details do not limitany invention or what is claimable in the specification but ratherdescribe features of the specific example embodiment. Features describedin the specification in the context of individual example embodimentsmay be implemented as a combination in a single example embodiment. Incontrast, various features described in the specification in the contextof a single example embodiment may be implemented in multiple exampleembodiments individually or in an appropriate sub-combination.Furthermore, the features may operate in a specific combination and maybe initially described as claimed in the combination, but one or morefeatures may be excluded from the claimed combination in some cases, andthe claimed combination may be changed into a sub-combination or amodification of a sub-combination.

Similarly, even though operations are described in a specific order onthe drawings, it should not be understood as the operations needing tobe performed in the specific order or in sequence to obtain desiredresults or as all the operations needing to be performed. In a specificcase, multitasking and parallel processing may be advantageous. Inaddition, it should not be understood as requiring a separation ofvarious apparatus components in the above described example embodimentsin all example embodiments, and it should be understood that theabove-described program components and apparatuses may be incorporatedinto a single software product or may be packaged in multiple softwareproducts.

It should be understood that the example embodiments disclosed hereinare merely illustrative and are not intended to limit the scope of theinvention. It will be apparent to one of ordinary skill in the art thatvarious modifications of the example embodiments may be made withoutdeparting from the spirit and scope of the claims and their equivalents.

What is claimed is:
 1. A control system using an in-vehicle gesture, the control system comprising: one or more processors; and memory storing instructions configured to cause the one or more processors to perform a process, the process comprising: recognizing the in-vehicle gesture according to sensing of a vehicle occupant; and selecting, from among functions invocable by the in-vehicle gesture, a function to be invoked by the one or more processors, wherein the function is selected based on the recognizing of the in-vehicle gesture and based on a sensed arm position of the vehicle occupant.
 2. The control system of claim 1, wherein the instructions are further configured to cause the one or more processors to select a first of the functions to be invoked for the recognized in-vehicle gesture when the sensed arm position corresponds to a first arm position and select a second of the functions to be invoked for the recognized in-vehicle gesture when the sensed arm position indicates a second arm position.
 3. The control system of claim 1, wherein the recognizing of in-vehicle gesture does not depend on the sensed arm position.
 4. The control system of claim 1, wherein the selecting the function to be invoked is further based on a classification of an emotion state of the vehicle occupant.
 5. The control system of claim 1, wherein the sensed arm position is based on an input from an armrest sensor of the vehicle.
 6. The control system of claim 1, wherein the functions include a first function and a second function, wherein the first function is selected for invocation by the recognized in-vehicle gesture when an arm-sensor senses an arm, and wherein the process further comprises: performing a second recognition of the in-vehicle gesture; and selecting the second function for invocation based on the second recognition of the in-vehicle gesture and based on the arm-sensor not sensing an arm.
 7. The control system of claim 1, wherein a first of the functions controls a user interface, wherein a second of the functions controls a vehicle function, and whether the first function or the second function is selected for invocation depends on the sensed arm position.
 8. The control system of claim 1, wherein one of the functions, or selection of one of the functions, is based on a determined emotional classification of the vehicle occupant that inputs the in-vehicle gesture.
 9. A control method for controlling vehicle functions using an in-vehicle gesture, the method comprising: responsive to a user input comprising video data captured by one or more vehicle cameras, recognizing the in-vehicle gesture from the user input; determining a state of an arm of the user; and selecting, from among the vehicle functions, a vehicle function for invoking, wherein the vehicle function is selected for invoking based on the recognized in-vehicle gesture and based on the determined state of the arm of the user.
 10. The method of claim 9, further comprising: receiving or determining a status classification of the user; and selecting the vehicle function based further on the status classification of the user, wherein the status classification and/or the status of the arm control which of the vehicle functions are selected for invoking.
 11. The method of claim 10, wherein the vehicle functions are respectively associated with user status classifications, and wherein the vehicle function is selected for invocation based on its being associated with the determined status classification of the user.
 12. The method of claim 10, wherein the status classification is an emotion classification.
 13. A method comprising: capturing, by one or more vehicle-compartment cameras of a vehicle, video segments of a hand of an occupant's arm while the occupant is in a seat of the vehicle, wherein a compartment of the vehicle contains the seat and an element; based on the captured video segments of the hand, performing respective recognitions of a gesture; each time the gesture is recognized, determining an arm status of the occupant's arm with respect to the element; each time the gesture is recognized and the correspondingly determined arm status comprises a first arm status: selecting a first function for invocation by the gesture; and each time the gesture is recognized and the correspondingly determined arm status comprises a second arm status: selecting a second function for invocation by the gesture.
 14. The method of claim 13, wherein the first arm status corresponds to the arm of the occupant having a first position or movement with respect to the element, and the second arm status corresponds to the arm of the occupant having a second position or movement with respect to the element.
 15. The method of claim 13, wherein the gesture corresponds to a pattern of hand movement in the air and the gesture is recognized based on the pattern of hand movement and not based on the determined arm status.
 16. The method of claim 13, wherein the first function controls an application, and wherein the second function manipulates a user interface.
 17. The method of claim 13, wherein the gesture comprises translation of the hand, and wherein, while the occupant is in the seat: the gesture can be performed and recognized when the occupant's arm is positioned in correspondence with first arm status; and the gesture can be performed and recognized when the occupant's arm is positioned in correspondence with the second arm status.
 18. The method of claim 13, wherein the first function comprises moving an image on a display according to the gesture, and wherein the second function comprises controlling content of the image according to the gesture.
 19. The method of claim 13, wherein an amount of hand translation of the gesture determines an extent or duration of an operation performed by the first function.
 20. The method of claim 13 wherein the element comprises an armrest. 